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Abstract. Over the past two decades, a large body of research has been conducted on the 
effectiveness of computer-mediated communication (CMC) employed as either stand¬ 
alone or instructional tools in SLA classrooms. Findings from this large body of work, 
however, are not conclusive, making it important to identify factors that would inform 
its successful implementations. This meta-analysis of empirical studies was conducted 
to examine the effects of CMC on language learning outcomes by calculating Cohen’s 
d effect sizes (Cohen, 1988) for each study on different learning outcomes. Altogether 
56 primary studies were retrieved as eligible studies between 2000-2011, including 27 
journal articles, 12 dissertations, 12 theses and 5 conference papers. Each study went 
through a two-level coding, the first level being study-feature coding and the second 
level effect-size coding. Twelve substantive and methodological features were coded for 
each study in the first level, 6 of which were identified as potential moderator variables 
that would affect the effectiveness of CMC differently. Tentative findings of this meta¬ 
analysis include: (a) there was a small to medium effect for CMC compared to face-to- 
face communication, (b) the effect of CMC was not equal for all language skills, and (c) 
small group sizes produced the largest effect compared to no grouping at all or groups 
with more than 3 students. 

Keywords: CMC, meta-analysis, SLA. 

1. Introduction 

Computer-mediated communication, defined as “multimodal, often (but not exclusively) 
Internet-mediated communication” (Thorne, 2008, p. 325), has been used extensively 
in second/foreign language classrooms since the late 1990s when the Internet became 
widely and immediately available for the general public, including educational sectors. 
CMC holds promise for language learning due to mounting evidence supported 
pedagogical benefits it brings to learning experiences. For example, the real-time nature 
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of the synchronous mode of CMC, such as Internet Relay Chat (chat room), creates a 
communication environment that simulates face-to-face conversation without visual 
clues. Via networked connection, CMC also extends a traditional language classroom 
to one that includes global communication involving a diversity of cultures if possible. 
Access to target language users, which used to be a mission impossible, is now as easy 
as a click of the mouse via “the instrumentality of computers” (Herring, 1996, p. 1). 
Through the 1970s into the early 1990s, the primary CMC tools were text-based; among 
them, email, Internet Relay Chat (ICQ) and MOOs (text-based virtual environments), 
were most popularly adopted in the language classrooms. With the advance of second- 
generation web applications, such as blogs, wikis and podcasts, the shortcomings of 
text-based CMC have been greatly overcome via advanced technologies that could 
distribute sound, video and varieties of media (Thorne, 2008). Meta-analysis has been 
used to integrate and compare the result of several studies since it was first introduced 
by Glass (1976). With the advancement of statistical techniques and controversial issues 
being mostly dealt with, meta-analysis has now become a preferred way of synthesizing 
research findings in scientific disciplines (Aytug, Rothstein, Zhou, & Kern, 2012). Over 
the past two decades, research syntheses conducted to answer questions such as how 
effective are technologies in promoting language learning were abundant; yet, specific 
syntheses particularly focusing on CMC remain scarce. 

1.1. Statement of the problem 

The integration of CMC into a language classroom remains to be a trend, and as the 
amount of anecdotes and empirical studies on CMC in language field recently increased, 
there is an urgent need for a valid conclusion to confirm our instinct that CMC does 
help language educators achieve intended results for their students. 

1.2. Research questions 

To enable precise analyses and interpretations of primary research findings, the study 
adopted a quantitative meta-analysis approach to synthesizing findings from empirical 
studies published between 2000 and 2011. Two overarching questions guided this 
research synthesis: 

• How effective is CMC in promoting second/foreign language skills (versus 
face-to-face communication or communication without computer/technology 
mediated devices)? 

• What are potential factors that mediate the effectiveness of CMC in promoting 
the acquisition of language skills? 

2. Methodology 

Empirical studies reviewed in this paper should meet the following criteria: 

• The study was published between 2000 and 2011; 
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• The study made use of some form of CMC (e.g., email, chat, conferencing, 
discussion forums, etc.) either exclusively or in conjunction with other 
instructional tools/methods; 

• The study addressed either the nature of the language produced during CMC and/ 
or the effect of CMC on L2 learning. (Both conditions required quantitative data); 

• The study employed an experimental or quasi-experimental design; 

• Studies recruited participants who were L2 or foreign language learners; 

• Studies included should report adequate quantitative information for effect sizes 
to be calculated. 

• For study reports across several sources, only one report was included in the 
meta-analysis. 

2.1. Coding scheme 

Each eligible study was coded at two levels: study level and effect size level. In the study 
level, coding, study characteristics, methodological characteristics and publication 
characteristics were coded (Lipsey & Wilson, 2001). Study level characteristics were 
further examined to decide whether they mediated the effectiveness of language 
learning outcomes. The entire coding consisted of three stages. In stage one, two coders 
independently coded each of the 56 primary studies in compliance with the coding 
sheet. In stage 2, half of the primary studies were selected randomly and codes were 
compared between the two coders. The initial inter-rater reliability was computed using 
the formula: number of agreed-upon codes over the total number of codes. 


Table 1. Coding scheme 


Characteristics 

Description 

Publication type 

Journal article/Book or book chapter/ Dissertation/thesis/Technical 
report/Conference paper (proceedings)/Other 

CMC Mode 

Synchronous/Asynchronous/Both 

CMC tool/platform 

Email/Chat/Discussion forum/Instant Messenger/Blog/E-portfolio/ 

Wiki/Other (specify) 

CMC Tool 

Voice/text 

CMC activity conduction 

In class/After class/Both 

CMC option 

Required activity/Optional activity/Other 

Group size 

1-5 people/6-10 people/11-15 people/More than 15 people/No grouping 

Length of experiment 

Short: Less than one month 

Average: between one month and four months (one semester) 

Long: more than one semester 

Outcome measurement 

Standardized achievement test (specify the title of the test) 

Teacher/Researcher developed achievement test 

Other (specify) 

Sample’s educational level: 

Elementary school level and below/Middle school level/College level and 
above 

Sample’s L2 

English/German/Japanese/Chinese/Spanish/Russian/Others 

Sample’s LI 

English/German/Japanese/Chinese/Spanish/Russian/Others 

Research design 

Pretest/Posttest control group design/posttest only control group design/One 
group only pretest and posttest design/Non-equivalent comparison group 
design 

Randomization 

Yes/No 

Reliability 

Internal consistency, test-retest correlation, etc. 
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Low interference characteristics such as the total sample size of the study, research 
setting and participants’ L2 were strived to reach the 100% inter-rater reliability. In 
stage 3, discrepancies in coding between the two coders for high-interference features 
such as the outcome measurement, treatment description, treatment duration, etc., were 
discussed and resolved. The final inter-rater reliability was 98%. Table 1 above provides 
the study-level coding scheme of major features in this meta-analysis. 

2.2. Outcome measures 

Treatment effects on four major language skills were measured and compared in this 
meta-analysis. Specifically, the outcomes included listening, speaking, reading and 
writing. Various competencies (e.g., grammatical, discourse, strategic, sociolinguistic 
competences, which make up specific language skills were classified into those specific 
skills. For example, studies that investigated grammatical or pragmatic competence 
were classified into the outcome category of writing. In the same vein, studies that 
investigated effect of CMC on pronunciation were classified into the outcome category 
of speaking. 

2.3. Effect size calculation 

The effectiveness of CMC on language skill acquisition was expressed by calculating 
effect sizes for each study on different learning outcomes. The effect sizes were 
calculated as the difference between the means of the treatment and the control groups 
divided by the pooled standard deviation of the sample, i.e., Cohens’ d. The potential 
outliers were checked for their influence on the overall mean effect. The effect sizes were 
also weighted/corrected for small sample sizes (Bangert-Drowns, Hurley, & Wilkinson, 
2004; Hoffler & Leutner, 2007). If a study did not provide descriptive analysis data but 
F values, effect sizes were calculated using the procedure suggested by Glass (1976). 

3. Results and conclusion 

3.1. Overview of eligible studies in review 

The final body of meta-analysis included 56 studies (data collection was completed by 
December 2011): 5 conference papers, 12 masters’ theses, 12 doctoral dissertations and 
27 journal articles. A total of 282 effect sizes were calculated from the 56 studies which 
contained a combined sample of about 3,713 participants. The sample size ranges 
from 12 to 354. 45 studies were published between 2006 and 2011; 11 studies were 
published between 2000 and 2005. Among the studies, 10 studies were carried out in 
middle schools, 2 in primary schools and the majority of studies were conducted in 
university/college settings (N= 44). In terms of research design, 49 studies included in 
the meta- analysis adopted a quasi-experimental design, using either a nonrandomized 
static-group posttest comparison design (5 studies), a nonrandomized one-group 
pretest-posttest design (16 studies), or a nonrandomized pretest-posttest control group 
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design (28 studies). Only 7 studies adopted a true experimental design with participants 
randomly being assigned to treatment groups and a true control group. English (44 
studies) is still the dominant target language of most CMC studies, followed by Spanish 
(9 studies), German (2 studies) and French (1 study). 

3.2. The effectiveness of CMC on language learning outcomes 

The results showed that of the 56 studies included, 79% (44 studies) of the study- 
weighted effect sizes were positive and favored CMC integration, while 21% (12 
studies) of them were negative and favored face-to-face or communication without any 
computer-mediated devices. 19 studies (34%) reported large effect sizes, 11 studies 
(20%) reported medium effect sizes and 26 studies (46%) reported large effect sizes 
based on Cohen’s (1988) interpretation guidelines of effect sizes. The overall effect size 
is .554, with 95% confidence interval between .482 and .626. Due to the limitation of 
space, moderator analysis results were not included in the paper; however, this meta¬ 
analysis found that task type, group size, participants’ educational level, CMC mode, 
CMC tool and outcome measures were potential moderators that would affect the 
overall effectiveness of CMC intervention. 
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